R and Quarto: A Foundation for Generating Reproducible Reports

Presentation to Ann Arbor R Users’ Group, Ann Arbor, MI

Steven J. Pierce

Center for Statistical Training and Consulting, Michigan State University

2024-11-14

Outline

What do you mean by reproducible?
Why should we aim for reproducibility?
How do we achieve reproducibility?

What do you mean by
reproducible?

Let’s define some concepts.

Reproducible Research (RR)

… is achieved when investigators share all the materials required to exactly recreate the findings so that others can verify them or conduct alternative analyses.

Statistical Results Can Be:

Repeatable

Original analyst
Original data

Reproducible

New analyst
Original data

Replicable

New analyst
New data

RR is a Product

RR is a product of how we work, not which methods we use.

Why aim for
reproducibility?

Several forces are promoting and enabling the push toward reproducibility.

Methods Matter!

Irreproducible < Reproducible < Replicated

Important

Reproducibility is an attainable minimum standard for science^[1].

Remember that our methods matter: how we conduct studies influences where our results fall on a continuum of trustworthiness and scientific credibility. Studies that use good scientific practices have higher trustworthiness and credibility. Let’s link this continuum to reproducibility.

At the low end, we have IRR, which relies on investigator reputation to establish credibility because key methodological details aren’t published and the findings can’t be independently verified. Reviewers and readers have limited ability to judge the quality of the research. IRR provides a poor foundation for future replication studies.

RR findings can be independently verified from the original data, reducing reliance on investigator reputation. Reviewers and readers can better judge quality of research on its own merits. Furthermore, RR provides a solid foundation for future replication studies by publishing key methodological details.

Finally, replicated research has the highest level of credibility because independent investigators have replicated the findings with new studies, indicating that the probability of opportunistic biases is low and the findings generalize beyond the original sample. This dramatically increases trustworthiness of a finding.

Roger Peng has argued that RR is an attainable minimum standard for science (Peng et al., 2006).

Funders Value RR

Data sharing and reproducibility initiatives

NIH logo. NSF logo.

IES logo.

Emerging Scientific Norms

Open science:
- https://www.cos.io/open-science
- https://open.science.gov
Pre-registered methods^[2–4]
Reporting guidelines^[5–8]
Publishing raw data^[9,10]
Journal & publisher policies^[3,9–12]
Institutional policies

Publishing Technology

Career Benefits of RR

Motivation to focus on quality
Become more efficient
Create more products
Easier to get published
Get cited more often
Build your reputation

How do we achieve reproducibility?

Understand the criteria, then apply principles, practices, & tools.

Criteria for Reproducbility
^[1]

Data
Methods
Documentation
Distribution

Materials Required to Recreate Findings

Materials	Findings
Manuals & procedures	Statistics
Instruments & scoring rules	Coefficients & p-values
Codebooks	Confidence intervals
Methods applied	Effect sizes
Data mgt decisions	Model fit indices
Data files	Figures
Software & analysis scripts	Tables

Principles for Achieving Reproducibility

Collaboration
Organization
Automation

Preservation
Integration
Separation

Understanding a few guiding principles helps people figure out how to do RR. Here are the principles I’ve identified so far. Thinking about reproducibility as collaboration encourages us to prepare and share materials that will make our work accessible to others. Organization simplifies communicating to others how to reproduce our work. Automation helps us achieve efficiency and accuracy because scripts can be faster and more reliable than manual processes. The preservation principle guides us to keep our raw data pristine and to keep a good audit trail. The integration principle suggests building documentation directly into data files, putting comments in code, and using code to directly insert statistical results into reports. The separation principle helps us manage complexity by splitting projects or tasks into pieces; for example separating data management from analysis, or keeping raw data separate from working data files used for specific analyses. Keep an eye out for instances of these principles in action throughout the rest of this talk. My examples will aim toward the middle of the complexity spectrum.

Workflow Woes

Use GUI, menus, & dialog boxes to do tasks
Manually updating data files
Version control through saving to new file names
Disorganized folders & files
No audit trail
Copy & paste output from stats software to word processor
Fixing mistakes took lots of time

As a student, I was taught a manual workflow for doing data analysis and writing reports. We used the GUI, menus, and dialog boxes in SPSS to do data management and analysis. We updated data files manually. We created tons of files, saving to a new filename as changes accrued. That led to confusing proliferation of folders and files with little planned structure. It was hard to tell who had last edited various files and which versions to use. We retyped or pasted results from SPSS output into Word documents then spent lots of time formatting them. If we made a mistake, it could takes hours or even days to re-do things. Over time I learned some strategies to improve that workflow, such as using SPSS syntax files to automate tasks, adding comments in the syntax files, adopting file naming conventions, and so on. But part of the problem was that the software tools weren’t integrated and lacked support for better workflows.

Spock

“Captain, you’re asking me to work with equipment which is hardly very far ahead of stone knives and bearskins.”

Star Trek (1966) - S01E28 The City on the Edge of Forever

Free, Open Source Software Tools for Reproducible Research

R, for statistical computing
RStudio Desktop, an editor
Quarto, for dynamic documents
Git, for version control
GitHub.com, for sharing & collaborating on code

Dynamic Documents Via Quarto + R

So what are dynamic documents? They draw on the concept of literate programming, where one embeds code directly in a document such as a report, interleaving it with narrative text, headings, and in-text citations. When you render dynamic documents, the code is executed and the results are directly inserted into the output. That includes statistics computed from the data, figures, tables, formatted equations, fully-formatted reference sections, and more. This diagram shows the files involved for a simple example. The idea is that rendering your report script produces a PDF file ready for delivery to your client.

Several principles are in action here: Separation of data from code, integration of R code into the Quarto script, where it automates data management and analyses. Quarto automates inserting results into the report. I’m using similar filenames to stay organized: they show which script creates the report. Saving all the files follows the preservation principle. One can rapidly reproduce the report by re-rendering the script, or fix an error in the script then render again to automate updating the report.

Research Compendium

A compendium organizes digital files so others can review or use them to reproduce results, or do new analyses.^[13] It should:

Use prevailing conventions to organize the project
Separate data, method, & output but show how they are related
Specify the software used in the original analysis

R Packages

… are folders of digital files designed for sharing code & help documentation.^[14] They

Follow a conventional structure
Can contain data, meta-data, & other documentation
Can contain dynamic documents & rendered output

Important

Use an R package for a research compendium!^[13]

Git Repositories

… are folders of files being tracked by Git for version control purposes.^[15,16] They:

Preserve a history of changes to the tracked file
Allow recovering a file’s prior state from the history
Can be private or public
Can be local or remote (hosted on a server)

Important

Put your research compendium in a Git repository!

GitHub

Collaboration

Treat RR as a form of collaborating with multiple stakeholders. Give them clear, useful, and relevant materials.

Who are your key collaborators?

Your future self
Your current team
Peer-reviewers
Your audience

Add diagram / photo illustrating collaboration with self, team-mates, and readers

Organization

Organize files into a research compendium.
Create documentation (README files, vignettes, etc.).
Use folder structures, naming conventions,
Use data flow diagrams and/or rendering scripts

Automation

Use scripts instead of GUIs
Make computers do the tedious labor
Computers are patient idiots

Preservation

What should you preserve?

Data and meta-data
Methodology decisions
Data cleaning, management, and analysis scripts
Key decisions and rationales
Source code for scripts
Version histories of data, code, output, and deliverables
Who changed what & when
Software environment & versions used

Integration

Attach meta-data directly to data
Include codebooks (or scripts to generate them from the data)
Integrate narrative text with code via Markup languages
Add custom functions and associated help files to research compendium
Include references

Separation

Data from code
Raw data from cleaned data
Code from output
Draft from production output
Version history from current files

Version Control

Git tracks how files changed (additions & deletions), who changed them, when, & why
Searchable & recoverable version history
Eliminates the need to use filename conventions for versioning

Narrative Text

Formatting
Headings
Links

Inline R Code

Tables

Figures

Reference Lists

Citation style language file to control format
BibTeX files containing data

Parameterized Reports

Map Your Path to Reproducibility

References

1. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789. https://doi.org/10.1093/aje/kwj093

2. Bosnjak, M., Fiebach, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., & Sokol, R. I. (2022). A template for preregistration of quantitative research in psychology: Report of the Joint Psychological Societies Preregistration Task Force. American Psychologist, 77(4), 602–615. https://doi.org/10.1037/amp0000879

3. DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015). Opportunistic biases: Their origins, effects, and an integrated solution. American Psychologist, 70(6), 499–514. https://doi.org/10.1037/a0039191

4. Moore, D. A. (2016). Preregister if you want to. American Psychologist, 71(3), 238–239. https://doi.org/10.1037/a0040195

5. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191

6. Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomized trials. Journal of Clinical Epidemiology, 63(8), e1–e37. https://doi.org/10.1016/j.jclinepi.2010.03.004

7. Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Journal of Clinical Epidemiology, 63(8), 834–840. https://doi.org/10.1016/j.jclinepi.2010.02.005

8. von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. PLoS Medicine, 4(10), e296. https://doi.org/10.1371/journal.pmed.0040296

9. Hrynaszkiewicz, I., & Altman, D. G. (2009). Towards agreement on best practice for publishing raw clinical trial data. Trials, 10(1), 17. https://doi.org/10.1186/1745-6215-10-17

10. Hrynaszkiewicz, I., Norton, M. L., Vickers, A. J., & Altman, D. G. (2010). Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. Trials, 11(1), 9. https://doi.org/10.1186/1745-6215-11-9

11. Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007). Reproducible research: Moving toward research the public can really trust. Annals of Internal Medicine, 146(6), 450–453. https://doi.org/10.7326/0003-4819-146-6-200703200-00154

12. Peng, R. D. (2009). Reproducible research and biostatistics. Biostatistics, 10(3), 405–408. https://doi.org/10.1093/biostatistics/kxp014

13. Marwick, B., Boettiger, C., & Mullen, L. (2018). Packaging data analytical work reproducibly using r (and friends). The American Statistician, 72(1), 80–88. https://doi.org/10.1080/00031305.2017.1375986

14. Wickham, H., & Bryan, J. (2021). R packages: Organize, test, document, and share your code. O’Reilly Media. https://r-pkgs.org

15. Bryan, J. (2018). Excuse me, do you have a moment to talk about version control? The American Statistician, 72(1), 20–27. https://doi.org/10.1080/00031305.2017.1399928

16. Chacon, S., & Straub, B. (2014). Pro Git. Apress Media. https://git-scm.com/book/en/v2

R and Quarto: A Foundation for Generating Reproducible Reports

Outline

What do you mean by reproducible?

Reproducible Research (RR)

Statistical Results Can Be:

RR is a Product

Why aim for reproducibility?

Methods Matter!

Funders Value RR

Emerging Scientific Norms

Publishing Technology

Career Benefits of RR

How do we achieve reproducibility?

Criteria for Reproducbility [1]

Materials Required to Recreate Findings

Principles for Achieving Reproducibility

Workflow Woes

Spock

Free, Open Source Software Tools for Reproducible Research

Dynamic Documents Via Quarto + R

Research Compendium

R Packages

Git Repositories

GitHub

Collaboration

Organization

Automation

Preservation

Integration

Separation

Version Control

Narrative Text

Inline R Code

Tables

Figures

Reference Lists

Parameterized Reports

Map Your Path to Reproducibility

References

What do you mean by
reproducible?

Why aim for
reproducibility?

Criteria for Reproducbility
^[1]